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We solve the largest sample eigenvalue distribution problem in 

the rank 1 spiked model of the quaternionic Wishart ensemble, which 

is the first case of a statistical generalization of the Laguerre sym- 

plectic ensemble (LSE) on the soft edge. We observe a phase change 

^^ ' phenomenon similar to that in the complex case, and prove that the 

^ , new distribution at the phase change point is the GOE Tracy- Widom 

• ■ distribution. 

-)— > 

1. Introduction. The Wishart ensemble is defined as follows [24]: 
Consider M independent observation xi , . . . , x^/ of an A^-variate normal 
distribution with mean and covariance matrix S. Here the values of the 
\^j \ normal distribution can be real, complex or even quaternion. If the variables 

pq ■ are complex or quaternionic, then the definition of the mean is as usual, and 

CN . the (co) variance is defined as 

t^ 

CN ; cov(x,y) =E((x-x)(y-y)*), 



where x (resp. y) is the mean of x (resp. y), and * is the complex or 
C^^ . quaternionic conjugation operator. Then S is a real symmetric/Hermitian/ 

quaternionic Hermitian matrix. Without loss of generality, we assume S to 
be a diagonal matrix, with population eigenvalues / = (/i, . . . , Z^r). If we put 
the above data into an A^ x M double array X = (xi : • • • :xm), then the 



o 



>< 



H \ positively defined real symmetric/Hermitian/quaternionic hermitian matrix 

S = -pXX* is the sample matrix and its eigenvalues A = (Ai,...,AAr) are 
sample eigenvalues. (X* is the transpose, Hermitian transpose or quater- 
nionic Hermitian transpose of X depending on type of X's entries.) The 
probability space of Aj's is called the Wishart ensemble. 
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It is a classical result [2] that (in the real category) if M ^ N, Aj's are 
good approximations of Zj's. But if M and A^ are of the same order of 
magnitude, that is, M/N = 7^ > 1 and M and A'^ are very large, the prob- 
lem is subtler. The simplest case with S = /, the white Wishart ensemble, 
is the Laguerre ensemble, well studied in random matrix theory (RMT) 
under the name LOE, LUE and LSE — they are abbreviations of Laguerre 
Orthogonal/Unitary/Symplectic Ensemble, and GOE, GUE and GSE ap- 
pearing later are abbreviations of Gaussian Orthogonal/Unitary/Symplectic 
Ensemble — over all the three base fields, respectively. 

Naturally, the next question is: If S is slightly deviate from /, such that 
li = l + ai, i = 1,. . . ,k, and Ik+i = ■ ■ ■ = In = ^, what is the distribution of 
the Aj's? This is called the spiked model [19] and k is defined as its rank. 

If M and A^ are very large and k and a^'s are small constants, the density 
of Aj's is the same as that in the white Wishart model, proved in [22] in real 
and complex categories. The distribution of the largest sample eigenvalue, 
however, may change. For the complex ensemble, Baik, Ben Arous and Peche 
[4] solved the problem completely. They show that if max(ai) is smaller than 
a threshold, then the distribution of the largest sample eigenvalue is the same 
as that in the white ensemble, which is the GUE Tracy- Widom distribution, 
but if max(aj) exceeds the threshold, that distribution is changed into a 
Gaussian whose mean and variance depend on max(aj). Furthermore, in 
the case that max(aj) equals the critical value, they find a series of new 
distributions, indexed by the multiplicity of max(aj). 

In the real category, which is practically the most important and math- 
ematically the most difficult, much less is known. In this paper I solve the 
distribution of the largest sample eigenvalue for the rank 1 spiked model in 
the quaternionic category. I believe the similarity of LOE and LSE [13] sug- 
gests that the solution to the quaternionic spiked model is an intermediate 
step toward the solution to the real one. 

1.1. Some known results for the largest sample eigenvalue in white and 
rank 1 spiked models. In latter part of the paper, we concentrate on the 
distribution of the largest sample eigenvalue in the rank 1 spiked model, so 
denote a to be the only perturbation parameter. 

The result in the complex category is complete. First we recall the result 
for the complex white Wishart ensemble. 

Proposition 1. The distribution of the largest sample eigenvalue in 
the complex white Wishart ensemble satisfies that, max(A) almost surely 
approaches [15] (l + 7~^)^ with fluctuation scale M~'^i^ , and [11, 18] 

Jnn^p(^(max(A) - (1 + r'f) ■ j^^:;^, <t)= Fgue^, 
where -Fgue is the GUE Tracy-Widom distribution. 
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The GUE Tracy- Widom distribution is defined by Fredholm determinant 
[11, 29]: 

Fgve{T) = det(l - KAiry{^,v)XiT,oo){v)), 
where X(t,oo) is the step function: 

X(T,oo)(??) = 



if ?7G (r,cx)), 
, otherwise, 

and -ffAiry(C)'^) is the well-known Airy kernel defined by the Airy function 

Ai(a;): 



(1) 



KMryit r]) = / Ai(e + t) Ai(r7 + t) dt. 
Jo 



The Airy function can be defined in different ways, and here we take an 
integral representation suitable for our asymptotic analysis [4]: 

(2) Am = ^[ e-^^^'/'^' dz, 

where r°° = Tf U Tf U Tf , which are defined as (see Figure 1) 



T^OO 
^ 1 



{-te^^/3|-oo<t<-l}. 



T-lOO 

^ 2 



{e- 



-iTTJI 1 



<i<|}, 



The breakthrough in the complex category is by [4] , which is for any finite 
rank spiked model. In the rank 1 case, it is: 

Proposition 2. In the rank 1 complex spiked model: 




Fig. 1. r° 
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1. //— l<a<7~^, then the distribution of the largest sample eigenvalue is 
the same as that of the complex white Wishart ensem,ble in Proposition 1. 

2. If a = 7~"^, then the limit and the fluctuation scale are the same as those 
of the complex white Wishart ensemble, but the distribution function is 



(3) lim ] 



'((max(A) - (1 +7-')') • 7^ 4/3 < t) = Fguei(T). 



(1+7)4/3 



3. If a> "f ^ , then the limit and the fluctuation scale are changed as well 
as the distribution function, which is a Gaussian: 



(4) 



lim pffmax(A)-(a + l)fl + ^)V- — ^i==== 



<r 



1 



27r 



e '"l^dt. 



The function -Fquei occurring in (3) is defined similarly to -Fque [4]: 

(5) Fguei(T) = det(l - (KAiry(e,^) + s('ne)Ai(r/))x(T,oo)(r?)), 

where s^^' is one of a series of functions defined in [4], and has the integral 
representation 



1 /* 1 poo 

3a)(^) = / e-^^+^/3^'-dz and s«(r/) = l-/ Ai(t) 

27ri Jr°° z Jn 



dt. 




Fig. 2. r° 
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where f °° = f 5° U f 2° U f 3° , which are defined as (see Figure 2 e is a positive 
constant, used later) 



1 =<-te 



^TTi/3 



oo<t<-|k fr = <^e' 



2]' ^2 l2 



1 5 

3 - - 3]' 



3 



- < t < CX3 



Remark 1. The kernel in (5) is not in trace class, but the Fredholm 
determinant is well defined and we can easily conjugate it into a trace class 
kernel. Several kernels below are in similar situations. 

In the real category, we have the result for the real white Whishart en- 
semble: 

Proposition 3. The distribution of the largest sample eigenvalue in the 
real white Wishart ensemble satisfies that, max(A) almost surely approaches 
[15] (1 + 7"^)2 with fluctuation scale M'^/a^ and [19] 

Jim^p((max(A) - (1 + ,-'f) • ^0^^ <t) = Fooe(T), 

where -Fqoe is the GOE Tracy-Widom distribution. 

Here the function -Fqoe is defined by the Fredholm determinant of a 
matrix integral operator [30]: 



i^GOE(T) = ^det(/- PGOE(e,r/)) 
and 



Pcoe(^,.) =X(T,oo)(0 (^,^^(^^^) - lsgn(x- ,) MV,^,) ) ^(--)(^)' 



2 

where 



Si(e,ry) = KAiry(C,r?) - ^ Ai(e) ^ Ai(t) dt + ^ Ai(0, 
SDiitri) = -^KAiryiCri) - ^ Ai{0 Ai{rj) , 

/■OO 1 /"OO /"OO 

ISl{C,r]) = -J KAiry{t,7j)dt+-J Ai{t)dtJ Ai{t)dt 

--J M{t)dt + -j A\{t)dt. 
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Remark 2. We have a more convenient form of Fqqe [12]: 

(6) i^GOE = v/det(l - (i^Airy (e, V) + s^'^ (6 Ai(r/))X(T,oo) (r/)), 

so [4] 

i^GUEl(r) = (FGOE(T))2. 

In the real spiked model, Baik and Silverstein [8] compute the almost sure 
limit of the largest population eigenvalue, which is the same as that in the 
complex category, and Paul [25] proves the Gaussian distribution property 
in the case a > 'y~^, which is similar to (4). Neither of their methods can 
find the distribution function when a<j~^. 

For the quaternionic white Wishart ensemble, we have: 

Proposition 4. The distribution of the largest sample eigenvalue in the 
quaternionic white Wishart ensemble satisfies that, max(A) almost surely 
approaches (1 + 7""^)^ with fluctuation scale M~^'^, and [I4] 

Jim^F((max(A)-(l + ,-)^).2(^<^ 

where -Fqse is the GSE Tracy-Widom distribution. 

Here the function -Fgse is defined by the Fredholm determinant of a ma- 
trix integral operator [30]: 



FGSE(r) = Vdet(/-P(^,r?)) 
and 

\ISi{i,Tt]) S4[rj,^,) J 

where 

M, V) = ^i^Airy (e, V)-\ Ai(0 I" Ai(t) dt, 



Id 1 



SD^i^, r?) = --—KMryit V)-- Ai(e) Ai{rj), 



ISi{i,ri) = --j KAuy{t,ri)dt + -j M{t)dtj k\{t)dt. 
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1.2. Statement of main results. The main theorem in this paper is: 

Theorem 1. In the rank 1 quaternionic spiked model: 

1. If —1 < a < 7^^, then the distribution of the largest sample eigenvalue is 
the same as that of the quaternionic white Wishart ensemble in Proposi- 
tion 4- 

2. If a = 7~^, then the limit and the fluctuation scale are the same as those 
of the quaternionic white Wishart ensemble, but the distribution function 
is 

3. If a > 'y~^ , then the limit and the fluctuation scale are changed as well 
as the distribution function, which is a Gaussian: 

Um F((ma^(X)-(a + l)(l + ^)] ^^^^ „ ., <t] 

= r -^e-'^'^dt. 

Here the function -Fgsei is defined by the Fredholm determinant of a 
matrix integral operator: 



i^GSEi(r) = Vdet(/-P(C,r?)) 
and 

\ISi{i,ir]) S4{V,^,) J 
where 

^4(e,r/)=54(C,r?) + ^Ai(0, TO(e,r/) = 5^4(e,r?), 

IS4{C,V) = IS4{C,V)-^J Ai{t)dt + -J Ai(t)dt. 

Although the distribution Fgsei seems to be new, we have that 

Theorem 2. 

i^GSEl(T)=FGOE(T). 
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1.3. Relation with other models and conjecture on the rank 1 real spiked 
model. The results of Theorems 1 and 2 give a phase transition pattern 
-^GSE~-^GOE^Gaussian as the parameter a increases from —1 to +oo. This 
pattern appears as hmiting distributions indexed by a parameter in several 
other combinatorial and statistical physical models, for example, the lengths 
of the longest monotone subsequences of random involutions with condition 
on the number of fixed points [6] and the symmetrized last passage per- 
colation [7] studied by Baik and Rains. In semi-infinite totally asymmetric 
simple exclusion process [26] studied by Prahofer and Spohn, and the sym- 
metric polynuclear growth process [5] studied by Baik et al., 2-dimensional 
phase transition diagrams are obtained, and the 1-dimensional -Fgse~-^goe~ 
Gaussian pattern is contained in both of them. 

Although there is no model which can give hints to the rank 1 real spiked 
model, it is plausible that it has a phase transition from i*GOE to Gaussian for 
the limiting distributions of the largest sample eigenvalue as a goes across 
7~^. Based on the duality of orthogonal and symplectic models from the 
Virasoro structure's point of view, we have: 

Conjecture 1. In the rank 1 real spiked model: 

1. If— l<a<7~^, then the distribution of the largest sample eigenvalue is 
the same as that of the real white Wishart ensemble in Proposition 3. 

2. If a = 7~^, then the limit and the fluctuation scale are the same as those 
of the quaternionic white Wishart ensemble, but the distribution function 
is 

3. If a > 7~^, then the limit and the fluctuation scale are changed as well as 
the distribution function, which is a Gaussian (proved by Paul in [25]): 



hm p('Lax(A)-(a+l)fl + ^')V- -^^E=== < T 

^ ~''/'dt. 



2tt 

1.4. Structure of the paper. In Section 2 we use combinatorial techniques 
to express the joint distribution function of {Aj}, and then by skew orthog- 
onal polynomial techniques express the distribution function of max(Aj) in 
the square root of a Fredholm determinant of a matrix integral operator. 
In Section 3 we do asymptotic analysis on the kernel of the matrix integral 
operator, and prove the three cases of Theorem 1 in the three subsections. 
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respectively. Section 4 contains the proof of Theorem 2. In the proof of The- 
orem 1, we use some trace norm convergence results which generalize the 
old result on the LUE [11], and we give a method of proof to them in the 
Appendix. 

2. The Fredholm determinantal formula. 

2.1. The joint distribution function. In this subsection, we prove the 
following: 

Theorem 3. The joint probability distribution function of X in the quater- 
nionic spiked model is 



(7) P(A) = iy^(A) J|(a2(^-^)+i.-2A/a 



In this paper, C stands for any constants, and here 

1 ■•• 1 

Ai 1 ■■■ Ajv 1 



V^X) = 



\i 2Ai ■ ■ ■ AjY 2Aiv 



^2N-2 (2iV-2)A?^-3 ... A^'^-^ (2iV-2)A^^-^ 



„a/(l + a)2MAi _a_2]^ „a/(l + a)2M \i ^ ^ ^ ^a/ (l + a)2M \j^ _a_2jyj-ga/(l + a)2A/Ajv 
1+a 1+a 

the determinant of a 2iV x 2N matrix whose (27V, 2fc- 1) entry is e"/(^+")2A'^^fc , 
(j, 2A; — 1) entry is A'^^ for j = 1, . . . , 2N — 1, and 2ith column is the deriva- 
tive of the (2i — l)st column. V^^(A) is a variation of the ^(A)^ appearing in 
the LSE (see [23] and (9)). 

For the Wishart ensemble defined in the introduction section, we first 
have the distribution function for the sample matrix in the N x N positive 
definite quaternionic Hermitian matrix space [3]: 

Remark 3. Due to the noncommutativity of the quaternions, detS* is 
not well defined in the usual way. Since S is quaternionic Hermitian, we 
can diagonalize it into a real- valued diagonal matrix by the conjugation of 
a quaternionic unitary matrix U, and define 

det "S = Jl eigenvalues of USU* . 

N 
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Remark 4. In the distribution function in real and complex categories 
of sample matrices, we do not need to take the real part of the trace, since the 
trace is already real. Unfortunately, this does not hold in the quaternionic 
category due to its noncommutativity, and luckily JRTr behaves better. [For 
example, 3?Tr(^5) = 3fiTr(SA), but Tr{AB) / Tv{BA) in general.] 

The distribution function for sample eigenvalues A, the eigenvalues of S, 
is 

1 ^ r 

(8) P(A) = l(y(A))4TTAf'"'^^^' / e-2MKTr(E-QAQ-)^g^ 



i=i 



where we integrate on the compact symplectic group with the Haar measure, 
V{\) = Yli^jiK — Aj) is the Vandermonde, and A = diag(Ai, . . . , Xn)- (See 
[23] for a derivation of the similar GSE case.) 

If the perturbation parameter a = 0, then /i = ^2 = • • • = /at = 1, 



jQeSv(N) . 1 



and 

N 

J 



(9) P(A) = ^{ViX)ril{Xf'-''^+''^-'^''^ 



is the standard LSE [23] . 
Generally, 

g-2MKTr(E-iQAQ-i) ^q 
QeSpiN) 

Jq&Sp{N) 

N „ 

^ TT g-2MA, / g2M3f{Tr((7-S-i)QAQ-i) ^q 

jLi JQeSpiN) 

Then by the integral formula of the quaternionic Zonal polynomials [17], we 
get 

f g2M3f{Tr((/-S-i)QAQ-i) ^q 

_ ^ {2My ^ c'(V2)(j-_^-i)g(i/2)(^^ 

^ j\ ^ ^(1/2)/. ^ 

J=0 -^ 1{k.)<N <--«; K^N) 
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where Ck {xi, . . . ,xj\f) is the N variable quaternionic Zonal polynomial, 
that is, the Jack polynomial with the parameter a = 1/2 (see [21] and [27]) 
and the C-normalization [10], so that [k = (ki, ■ ■ ■ ,ki), ki>k2> ■ ■ ■ >ki> 0, 
then 1{k) = I] 



J2 Cj^^'^\xi,...,x^) = {xi + --- + x^ 



k 

m) ■ 



l{K)<m 
Khk 



In the formula, a symmetric polynomial of a matrix is equivalent to the 
symmetric polynomial of its eigenvalues, so 

Since all variables except for one vanish in Ck {I — S~^), we simply find 
(12) C(^/'\l-^-%,^^,= 0. 

We have [27] 

crfT^.o 



'(J) Vl+a' J Vl+O 

and since the number of variables is N [27] 

so with (11) and (12), we get 

g2MKTr((/-S-i)QAQ-i) ^g 



QeSp{N) 

oo 



(2Myc;;f'(/-E-)c;;f'(A) 



?! /^(l/2)/r N 



niZom+i)\i+a-"i -^0) 



3=0 

In [27] there is an identity 

N 
Comparing it with the well-known identity for Schur polynomials 

oo N 



oo N -. 



7=0 7=1 J 
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we get the identity 

(13) (j + 1)C{J(')(A) = S(,)(Ai, Ai, A2, A2, . . . , A^, Xn), 

with each Aj appearing twice as variables of the suy For notational siniphc- 
ity, we denote the right-hand side of (13) as S(j)(A), which is a plethysm 

[21] 



Now we get 
(14) 



/ 

Jc 



QeSp{N) 



S(j)(A) = S(j)o2pi(A). 

j2M»Tr((/-S-i)QAQ-i) ^q 



E- 



1 



a 



Then we need a lemma to simplify (14) further. 



2M S(,)(A). 



Lemma 1. 



HJ) 



(A) 



(15) 



1 

Ai 
A? 




1 

2Ai 



^2N^2 (27V-2)A2^-3 

X V{X)-\ 



1 
Aiv 





1 

2Ajv 

A?f -2 (2iV - 2)A?f -=* 

A?f+^-^ (2iV + j-l)A^^+^- 



wzi/i the {k,2j — 1) eniry of the matrix being a power of Xj with the exponent 
k — 1 if k ^ 2N and 2N + j — 1 if k = 2N , and the {k, 2j) entry being the 
derivative of the {k,2j — 1) entry with respect to Xj. 

To prove this lemma, we need the well-known fact (see [23]), proven by 
L'Hopital's rule 



(16) V{XY 



1 

Ai 





1 



vf-i 



^z.v-i (2Ar_i)Af 



2N-2 



1 

Aat 



^N 





1 

(2Af-l)A 



N 



with the {k,2j — 1) entry being A- and the {k,2j) entry {k — 1)A- 

Proof of Lemma 1. Applying the L'Hopital's rule repeatedly with 
respect to X2i, i = l, . . . ,N, we get the identity 
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1 

Ai 1 

A? 2Ai 



(2Af-2)A^"-3 
(2Af + j -1)A^"+^" 



A^"-i (27V-1)a2 



/ 



d 



N 



dx2 0X4 ■ ■ ■ dx2N 



V 





1 



A?f-^ (2iV-2)A^/-^ 

A^/+^-i (2JV + ,-l)A^"+^--^ 

1 

An 1 



{2N-l)\j 



1 

xi 



1 

X2 



2N-2 2N-2 



2JV+J-1 2N+J-1 



1 

a;2iv-l 



2iV-2 
^2N-1 



1 

a:2JV 



2iV-2 
^2N 



2JV+J-1 27V+J-1 

^2iV-l ^2iV 



QN 



8x2 9x4 ■ ■ ■ dx 



2N 



1 




1 


xi 




X2 


xl- 


2 


2JV-2 
^2 


2N+3 


-1 


^2iV+3-l 



1 1 

X2N — \ X2N 



2JV-2 
''2JV-1 



2JV-2 
''2JV 



2JV+J-1 2JV+J-1 



■ ■S(i)(-^l7-^l)'^2, A2,. . . ,Xn,Xn) = ^(j)(^)) 



2:21-1 = a;2i = A, , 
i = l,...,N I 



from the matrix representation of Schur polynomials, and now use (16) to 
get the compact formula (15). D 

Substituting (15) into (14), we get 



v{\y I 



2MKTr((/-S-i)QAQ-i) 



clQ 



(17) 



1 

Ai 
A? 




1 

2Ai 



■^2N-2 (2iV_2)Af 

p(Ai) p'(Ai) 



1 

Ajv 

Ajv 




1 

2Ajv 



A^'^-^ (2Af-2)A^^-3 
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1 

c 



1 

Ai 

A? 





1 

2Ai 



{2N - 2)XY 



ga/{l + a)2Al"Ai a njy^^a /a + a'l2A/Ai 

1 + a 



c 



y^(A), 



1 

Ajv 





1 

2Ajv 



^2N-2 (2Ar-2)A^^-3 

ga/(l + a)2MAjv _2_ 2Me''/(l + '''^-*-'^-^« 
1 + a 



where 



^(^) = EffFT 



om=o(2iV + OVl + " 



(2iV-l)! 



(a/(l + a)2M) 



2Ar-l 



2M x^^+^-i 



2N-2 

E 

j=0 



„a/(l+a)2Afx 



1 



i!Vl + a 



-2Mx 



v~ 



and if fc 7^ 2N, the (/c, 2j — 1) entries in both matrices are A ■ , and the 
{k,2j) entries are {k — 1)A~ , and the 2N,2i — 1 entry in the former (lat- 
ter) matrix is p{Xi) (resp. e"/(-^+")^^^-^» ) and the 2N,2i entry p'{Xi) (resp. 

Proof of Theorem 3. Formulas (8), (10) and (17) together give the 
result (7). D 

2.2. The Pfaffian and determinantal formulas. With the formula (7) 
ready to use, we apply the standard RMT technique to get the distribu- 
tion formula for the largest sample eigenvalue, in the same spirit as the 
solution of the LSE. Our process below is closely parallel to that in [31] to 
the LSE. 

First, we find a skew orthogonal basis {ipo{x),(pi{x), . . . , (p2N-i{x)} of the 
linear space spanned by {l,rE,a;^, . . . , j;^^~^,e'^' (^~^'^)^*'^^}. We require that 
the ^j{x) is a linear combination of {l,x,x'^, . . . ,x^} if j < 2N — 1, while 
f2N-iix) can be arbitrary, with the skew inner products among them 



{ipj{x),ipk{x))4 



{ipj{x)ip',{x)-^'Ax)Mx))x^(-''-''^+'e-'''^dx 



-rk/2, 



if j is even and k = j + 1, 
if k is even and j = k + 1, 
otherwise. 
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P(A)4 



'/5o(Ai; 
93i(Ai; 



V'UAi) 









(18) 



V32Af-i(Ai) V'W-ilAi) 

N 



xn(A? 

^o(Ai 
V'i(Ai 



2(A/-Ar)+l^_2A/A,' 



1 

c 



^o(Ai] 
V'KAi] 



'/'27V-l(AAf) 992^_]^(AAr) 



V'o(AAr) V'o(AAf) 

V'l(Aiv) V'ilATv) 



V'2Ar-l(Al) V'W-llAl) ••• V'2Af-l(AAr) '02Af-l(Ajv) 

where 

(19) V',(x)=^,(x)x*^-^+i/2^-^^^ 

For an arbitrary function f{x) on [0, oo), by the formula of de Bruijn [9], 
V'o(Ai) V'o(Ai) ••• V'o(A^) ^^(A^) 

V'i(Ai) i;[{Xi) ••• Vi(Aiv) V'UAiv) 



■oo /-oo 



JO 



(20) 



V'2Af-l(Al) '02Ar_i(Al) ••• V'2Af~l(AAr) V'2Af-l(AAf) 



N 



n(l + /(A.))dA, = CPf(P(l + /)), 



j=l 



where P(l + /) is a 2N x 2A^ matrix, whose entries depend on 1 + / in the 
following way: 






Now we define a matrix Z as 
/ ro 
-ro 



ri 
-n 



with 



rk/2-i, 
0, 



r7v_i 

-r^-i / 

if k is even and j = k — 1, 
if j is even and k = j — 1, 
otherwise, 
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and define for j = 0, . . . , A^ — 1, r\ = Z ^^, that is, 

ry.,(x) = -^;?^±iM and ,,,,,(.) = M^. 



So we liave 



/■oo 



= Zj^k + / (^j--i(x)^fc„i(x) - i>]_^(x)^\}k~\{x))^ {x) dx. 
Jo 

And if we denote Q(l + /) = Z-ip(l + /), then 

/■oo 

(3(1 + /)i,fe = (^j.fc + / {Vj-i{xWk-iix)-Vj-i{x)i'k-i{x))f{x)dx. 



If we choose / to be — X(t,oo) ) then the integral on the left-hand side of 
(20), after multiplying a constant, is the probability of all Aj's smaller than 
T. In latter part of the paper, we abbreviate X(T,oo) to x- So we get for a 
T-independent constant 



(max(A,)<T) = CPf(P(l-x)), 



and 



(P(max(A,) < T)f = C^ det(P(l - x)) = C2det(Q(l - x))- 

In linear algebra, we have the determinant identity 

(21) det{I-AB) = det{I-BA), 

for A an mxn matrix and B annx m matrix, but the identity still holds in 
infinite dimensional settings [16]. Letting det mean a Predholm determinant 
for matrix integral operators, we describe a setting due to Tracy-Widom 
[31]. 

If A is an operator from -^^([0, oo)) x L^{[0, oo)) to the vector space M? 
with 



A 



9{x) 



oo poo 

X{x)r]j^i{x)g{x)dx- / x{x)r]'j_i{x)h{x)dx, 
Jo 



and B is an operator from M^^ to L^([0,oo)) x L^([0, oo)) with 



2N 



B 



c\ 



S2N , 



^Ck'il)'k^i{x)x{x) 

k=l 
2N 

^Ck'4]k^i{x)x{x) 



^k=l 



QUATERNIONIC WISH ART 17 

then 

I-AB = Q{l-x) 
and 

-^^ ^\ISi{x,y) Si{y,x) J^"-^'' 

where S4{x,y), IS4{x,y) and SD/i{x,y) are integral operators whose kernels 
are 

2N-1 

S4{x,y)= Y, i^'ji^Hiy) 

j=0 

N-l , 

= Y. —i-'^2ji^)'^2j+iiy) + ip2j+iix)iJ2j{y)), 



(22) 



2N-1 

SDi{x,y)= Y -i'jixWjiy) 

j=0 



N-l^ 



J2 —ii^2jix)^2j+iiy) - i'2j+iix)i^2jiy))^ 



j=o ""j 



2N-1 

ISi{x,y)= Y '4'j{x)i]j{y) 

j=0 



N~l I 



Y —i-i'2jix)ip2j+i{y) + ip2j+i{x)ij2j{y)), 



3=0 ""^ 



(23) 



2N-1 

S4{y,x)= Y -i^ji^Wjiy) 

j=0 



N-l -. 



J2 — ('^2i(x)V'2j+i(y) - V'2j+i(a;)V'2j(2/))- 



j=o ""j 



Remark 5. It is clear that the nomenclature of SD4{x,y) is due to the 
fact that SDi{x^y) is the negative of the derivative of S'4(x,y). But IS4{x,y), 
which gets its name in the same way in earlier literature in GSE (e.g., [30]), 
in our problem may not satisfy the equation 

ISi{x,y) = - S4,{t,y)dt, 

J X 

since the integral on the right-hand side may diverge. 
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In conclusion, 



and we can find that C^ = 1 by taking the hmit T — > oo. We define a 2 x 2 
matrix kernel as 

x{x)S4{x,y)x{y) x{x)DS4{x,y)x{y) 
x{x)IS4{x,y)x{y) x{x)Si{y,x)x{y) 



then we have 



(P(max(Ai) < T)f = det(/ - Pt{x, y)). 



2.3. S4^{x,y) in terms of Laguerre polynomials. In manipulation of skew 
orthogonal polynomials, we take the approach of [1], and all classical or- 
thogonal polynomial properties are from [28]. 

Since Laguerre polynomials by definition satisfy the orthogonal property 

'^LfL^^^x-e-^dx=^-l±;^S„„ 

^ r- 

and they have the differential identity [we assume Ln (x) = if n < 0] 
(24) x^lW(x) =nL(r)(x) - (n + a)Lt\(x), 

it is easy to get that 

^^(2(M-iV))(2Mx),4^(^-^-^»(2Mx))4 

= r fif ^''-^^^(2Mx)^Lf *^-^))(2Mx) 
Jo \ ■' ax 

, 2(M-7V)+l(^+2(M-iV))! .„. . ^ 

. ;L \2(M-JV)+i(^^2(M-iV))! 

— : r: , ifA;=7 + l, 

\2Mj {k-iy. 

, 0, otherwise. 
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So we can choose for j = 0, . . . , A^ — 2, 

(^^) ..(^)-|:(n,f^|s^)4r-"''(2M.). 

(26) V2j+i(a!) = -LgV*;-"''(2Mi) 

and 

ro7l =fJ-\ '("'-'^)+' (2j+2(M-JV) + l)! ^ 2k-l 

^ '^ ''i \2m) (2j)! ^^-^ 2fc + 2(Af - iV) ■ 

We can also choose 

but ip2N-i{x) is not a polynomial and needs to be treated separately. 
By the Rodrigues' representation 

and repeated integration by parts, we get for n > 

^ga/(l+a)2Mx^^{2(A/-^))(2Mx))4 

^^. 2(M-7V)+Y +i(n + 2(M-^VH^ 

2My' l^ "^ n! 

_ i (n + 2(M-iV))! 

^ ^ (n-1)! 
and 

SO that 

l + Q^2(A/-JV)+i^^.^^ (2j + 2(A/-iV) + l)! -j^ 2fc-l 



a 



2M ; (2j) ! ^J-^ 2fc + 2(M - iV) 

and 

•l + a^^2(A^^-^)+7 ,^.^, (2j + 2(M-JV) + 2)! 



o 



2M y V (2i + l)! 

2, (2j + 2(M-JV) + i)! 

~ o ; — 

(2j)! 
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Now by the skew orthogonality, we can choose 

N~2 



V,2^„i(x) = e'^/(i+'^)2*-^-- ^ l((e'^/(l+'^)2*■^^V,2,4-l(x))4^2,(x) 



j=o ""j 



{e'''^'^''^^^\^2A^))mj+i{^)) 



and 



2Ar-2 
j=0 






2M y (2iV - 2)! ^J-^ 2k + 2(M - N) ' 

Now, we write S4{x,y) as 54a(x,y) + S4b{x,y), where 



N-2 . 



(28) S4aix,y) = Y^ — (-^2i(a;)V'2j+l(y) +^2i+l(^)V'2j(y)) 

and 



i=o '^J 



(29) S'4fe(x,y) = {-tp2N_2{x)'ll)2N~l{y) + 1p2N-l(.^)i^2N-2{y)), 

TN-l 

and simplify them separately. 

The formula (28) of our S^aix^y) is also the formula for S4{x,y) in the 
LSE problem, with parameters M and N — 2, and has been well studied. For 
completeness we derive its Laguerre polynomial expression here, following 

By the differential identity (24) and the identity 

nL^^"\x) = {-x + 2n + a- l)4"~i(^) " (n + a - 1)4-2 (^)> 
we get, remembering the definition (19), the telescoping sequence 



^fM-N + l/2-Mx + x^) Lg^^-""^^ {2Mx)\ 



X ajA^-^'-i/Sg-Mx 
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j k 



(30) 



1 yrr — ^]-^ r(2fc + i)4?*f-^»(2Mx) 

2^^y 2i + 2(M-Ar)^^ ^ 

- {2k + 2(M - iV))4fc^{"^^^ {2Mx)) 

ifn ^^^-^ ] 

2\^}^2k + 2iM-N)J 

X (2j + l)Lg.(i'-'^»(2Mx)x^^-^-i/2e"^^ 



and 



/2-Mx + x^)4'^^^-^" 



V'2i+i(3;) = - (^Af - Af + 1/2 - Mx + X— )^2j+i 
X (2Mx)x^^-^"i/2g-^'^^ 
(31) =-l((2j + 2)45V1"'^))(2Mx) 

- {2j + 2(M - iV) + 1)45^*^"^^^ (2Mx)) 

Therefore, if we substitute (27), (30) and (31) into (28), we get after some 
trick, 

S4a{x,y) = l(2M)2(A'^-^)+ix*'^-^-V2e-^^-yA^-^+V2e-Afy 



■2N-2 



,-l 



x<| y -21 -Lf(^^-^»(2Mx)Lf(^-^-^»(2My) 

_ {2N - 2)! /^^ 2j + 2(M - N)\ 
~ (2M-2)! [^}^ 2^1 y) 

x4'^^2"'^))(2Mx)^2JV-2(2/)|. 

Furthermore, we can simphfy ip2N~2{x)- Since for j ^ 2N — 1 [if we define 
c/5j(x) and then '4'j{x) for j > 2N — 1 by the formula (25) and (26)] 

oo 

(V'2Ar-2 {x)lp', (x) - '02Af-2 (a^)^j {x)) dx = 0, 


we get for j ^ 2N — 1, using integration by parts, 

' V'27V-2 (x)Lf ('"-^» (2Mx)x^'^-^+i/2e-^^- dx = 0. 
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So by the orthogonal property of Laguerre polynomials, we get 

and we can determine that 

2JV-1 ^^ 2j-l 

2 |J 2j + 2(M - iV) 

without much difficulty. Together with the fact lmix^ooi^2N-2{x) = 0, we 
get 

^^^-^("^ = Y- n 2j + 2{M-N) 

Now, we can write S4a{x,y) as 54ai(x,y) + S4a2ix,y), where 

54al(x,y) = ^(2M)2(^^-^)+l 



X if "-"»(2Af!,)»"'-"+"=e-"'!' 



and 



S,a2{x,y) = l(2M)^(M-7V)+i (2A^-l)! 
^ ^ 4^ ^ (2M-2)! 

(33) X 4';Ji'2"'^" (2Mx)x^^-^"i/2e-^- 






1^ 2Af ^2(^^"^)+\_(2JV_i) (2iV-l)! 



Finally, 

^''^''^y'^ = -2[ff-a) " (2M-1)! 

(34) X {Lfi'!-''^\2Mx)x''-^-'/\-^^^i;,^^,{y) 



+ i^',j,_,{x) / 42ii\"^»(2Mt)t^-^-^-V2e-^^*dt 

Jy 

and we can take the asymptotic analyses of 5'4ai {x, y) , Sia2 {x, y) and S^b{x-, y) 
separately. 
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3. Asymptotic analysis. In order to consider the rescaled distribution 
problem, we wish to find the probability of the largest sample eigenvalue 
being in the domain {0,p + qT]. We can put the kernel in the new coordinate 

system [after a conjugation by C^ „ _i/2 )]> and get 

(P(max(A.) <p + qT)f = det (l - x{x) ( ^'^^^^"^^^ ^£^^^g) ) x{^)] 

= det(/-PT(e,^)), 
where as L^ functions, 

(35) SD4{^,r])=q^SD4{x,y)\a:=p+q^, 

y=p+qrj 

(36) -5*4(^,7/) =g54(x,y)U=p+g^, 

y=P+qv 

(37) IS^i^v) = IS4{x,y)\^=p+q^ 

y=p+qri 

and 

In this section, we want to prove that for fixed 7 > 1 and o > —1, we can 
choose suitable pM and qm depending on M, so that for any T, 

lim (P(max(Ai) < pM + (?A/T))' = lim det(/ - Prit v)) = /a(T), 

where fa is a function to be determined. 

To prove the convergence of Fredholm determinants, we may use that 
Pt{C,t]) is in trace class for any M and converges to a certain 2x2 matrix 
kernel in trace norm. Equivalently, we may use that each entry of Pri^jV) 
is in trace class and converges to a scalar kernel in trace norm. It turns 
out later that the Pt('^,??)'s may not satisfy these requirements, but certain 
conjugates do. 

Since the IS4{x,y) and DS4{x,y) are of the same form as S4{x,y), we 
only show the asymptotic analysis of S4{x,y), and state the result for the 
other two, for which the arguments are the same. 

3.1. Proof of the —1 < a < 7"^ part of Theorem 1. In case — 1 < a < 7"^, 
we choose pM = (1 + 7"^)^ and qm = (TmW^^ ^^^ denote [here * stands 
for 4, 4a, 4al, 4a2 and 46; the definition of S^:{S,,rj) in (38) is only used in 
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Sections 3.1 and 3.2] 

_ (1 + ^)4/3 

(38) S^{C,r]) = ,2jvf)2/3'^*'^^'2^)l^-=(i+7-i)2 + (i+7)*/V(7(2M)2/3)^ • 

j;=(l+7-l)2 + (l+^)4/3/(^(2M)2/3)r, 

S4,a{x,y) is the formula for the upper-left entry of the 2x2 matrix kernel 
of the LSE problem with parameters M and A^ — 1, and its asymptotic 
behavior is well studied [14]. We want to prove that as M— >oo, Sia{x-,y) 
dominates S4{x,y) in the domain that we are interested in, and so naturally 
the distribution of the largest sample eigenvalue in the perturbed problem is 
the same as that in the LSE problem. (The difference between N and A^ — 1 
is negligible.) 

S4:ai{x,y) is almost the kernel for the LUE problem with parameters 
2M — 2 and 2N — 2, besides a factor ■\/y/x/2. From a standard result for 
LUE [11], XTiO'^iaii(,,v)XT{ri) is in trace class and converges in trace norm 
to half of the Airy kernel 

(39) hm x{OSAai{^,r,)x{v) = \x{OKAny{C,ri)x{v)- 

More discussion see the Appendix. 

For the Sia2{x,y) part, we also have in trace norm [14], 

(40) hm x{i)SAa2{i,r,)x{il) = -]x{0^m r ^mdtxiri). 

We just sketch the proof. Since Sia2i(,,'n) is a rank 1 operator, for the trace 
norm convergence, we only need to prove that in L^ norm as functions in ^ 
and respectively r], 

hm 7-2^(1 + 7)^/3(2M)i/3e^-^ 

M— >oo 

(41) X 4'ii'2^'^» (2Mx)rE^-^-i/2e-^^^x(e) 
= Ai(Ox(e), 



(42) 



00 



2^9A/rpAf-A^ / r(2(^^-^))r9A/f^VA^-^"V2,-Aft 



hm 7-^^^2Me^"^^^ / L^^V (2^*)* ^ ^^ e-^"*dtx(r?) 

M^OO Jy 

Ai{t)dtx{v) 



V 



and by the Stirling's formula. 



hm (2M)2(^^-^)'i (2A^- 1)! 2(JV^M) 4iv-i ^ ^ 
M^oo^ ^ (2M-2)! ' 
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By (33), (38), (41) and (42), we get 

X (1 + 7)^/3(2M)i/3e*^-^42^i'2"'^^^(2Mx) 
X r ^'i-'r"^" (2Af i)t^'^^-^/'e-*« dtxin)- 



•'y 

Therefore we get the trace norm convergence from the L^ convergence by 
the fact that if fn{x) -^ f{x) and gn{y) -^ g{y) in L^ norm, then we have 
the convergence of integral operators in trace norm: 

fn{x)gn{y)^ f[x)g{y). 

Finahy, we need to analyze the term S^bH-, ^)) new to the perturbed prob- 
lem. We need the following results: 

Proposition 5. For fixed 7 > 1 and —1 < a < 7^"*^ and any T, we have 
the convergences in L^ norm, with respect to ^ or rj: 

hm 7-2^-1(1 + 7)4/3 (2A/)i/3eAf-7V 

A/— ♦CO 

X 4'^i'-^»(2Mx)x^^-^-i/2e-^-x(e) 

= -Ai(e)x(o, 

hm 7-2^2Me*^-^ H L^^i''-''^\2Mt)t^'-^~'/^e~^'' dtxiv) 

M^OO Jy 

) 

Ai{t)dtx{v), 

(43) 

= Mv)x(.ri), 

j,m^(i + .)2(«-«)-.„--« (l_j^|rti)!::!e«-«4„_,(.),(5) 

= Ai'(Ox({). 

Proof. We just prove the identity (43), and others can be done in the 
same way. 
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By the integral representation of Laguerre polynomials, 

(44) .fM-«„„M.)^^£f ^±1^ ., 

where C is a contour around the pole 0, therefore we get 

(l + Q)2(Af-JV)+l 

2m 
/c l + a{z + l)/z z 



(45) 



27rz Jc z + a/{a + l) 

27ri 



Tc^ ^27V ((a + i)2 + o)(z + l) 

If the pole z = — ^ty is inside of C, then 

27ri Jc -z + a/{a + 1) 



and 



(46) 



v^2^-i(y) = ^- 



X (J) e-2My^(i±l)^ f , 

c ^27V ^^a + l)z + a){z + !)'''■ 

In later part of the proof, we make this condition hold, and will not mention 
the canceled terms, and we are then free to deform C in (46) as we wish, 
provided it includes 0. We then proceed to a stationary phase analysis. 
Since 

-2Myzi^±}f^ _ 2M(^{l+'f-^fz+log{z+iy-,-^logz)-^^±^^{2M)^/'^riz 
^ ^2N ^ 

(we do not need to concern ourselves about the ambiguity of the value of 
logarithmic functions), if we denote 

(47) f{z) = -{l+j-^fz + log{z + l)-j-Hogz, 

then we get: 



/"(-4t) = o; 
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with the zero point z = — -^ ; 
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7+1 

So locally around z 
1 



1 

"7+1- 



/ 



(48) 

where 

(49) 



7 + 1 



+ w 



7 + 1 



+ log7-(l-7 ^)log(7 + l) + 7 ^vri 



+ 



(7 + 1) 
373 



-w'^ + Ri{w), 



Riiw) = 0{w^), 



as 1(7 — > 0. 



After the substitution w = z + -^ , we get 



^2A/(-y2;+log(z-l)-7-2logz)_ 



C 



{{a+l)z + a){z + l) 



dz 



j> exp|2M 



7 + 1 



+ 



(7 + 1)' 



+ log7-(l-7 ^)log(7 + l)+7 ^TTz 



w^ + Ri{w) 



(1+^)4/3 



373 "^ ' "''"'^ 7(2M)2/3' 

w- 1/(7 + 1) 

((a + l)u; + (a7 - l)/(7 + 1))(«; + 7/(7 + 1)) 



rj\w 



7 + 1 



dw 



1 



7 



2Af-l 



a+l(^ + l)2(A/-JV) 

(1 + ^)4/3 



„2A//(1+7)S/ 



X 0) exp 



7 



(2M)^/37?u; + ^^^l^ 2Mw^ + 2MRi{w) 



373 



-(7 + l)it; + l 



■ diu, 



(7 + l)/7«; + lw + {a'y- l)/((7 + l)(a + 1)) 

where F*^ is a contour around — ry, composed of Ff^, T2^ , T^,^ and r4^, 
which are defined as (see Figure 3) 



pA/ 
'- 1 



-.A/ 



(4-t) 



^_e-V3 
7 + 1 



0<t<4 



:i + 



:^(2Af)- 



-1/3 



7 



(1 + 7)4/3 



(2M)- 



-1/3 -twi 



1 1 

-<t< - ,, 

3 - - 3]' 



F 



M 



|t^— e^"'/^ ^^(2M)-^/3 < t < 4I 

I 7 + 1 (1 + 7)^/^ " " J 
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Fig. 3. 



'-4 



7 



7 + 1 



+ it 



-2^3 



7 + 1 



<t<2\^ 



For the asymptotic analysis, we define 



-.M 



(50) r^^U = {zGr^^^|3?(z)<(2M)"^'^/^^}, r 



10/391 



7 + 1 



remote 



(rf urf )\r 



M 
local ' 



(51) T% = {weT'=^\^{w)<c}, rg; = r~\r^^. 

Now, we denote 

(1+^)4/3 



FaM{ri,w) 



X expj- ^ ^^^ — {2Mfl^r]w + ^ ^1^ 2Mw^ + 2MRi{w) 



7 



-(7 + l)'u; + l 



37^ 



1 



(7 + l)/7u; + 1 u; + (07 - l)/((7 + l)(a + 1)) ' 
and establish several lemmas for the proof. D 

Lemma 2. IfT is fixed and M is large enough, then for any r]>T, 



1 
27ri Jr^' 



FaM{'n,w)dw 



< 



3 Ml/40 • 



Proof. By (48) and (47), 

, -2Mw^ + 2MRi(w) 



(1+7)' 



37^ 



(52) 
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log7 + (1 - 7"^) log(7 + 1) - 7~^7ri 



ZM I ?; w + log 1 w + 1 
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r 



7 



— 7 log((7 + l)u; — 1) — 7 vri j . 
U w£ Tj^, ^{w) = 2-^, and denote 9 = arg(w) G [-|, |], we have 



^i '^tl 2M-u;^ + 2MRi{w) 

:2M( -2^^^ + log(-y/32 + (2 tan 0)2) 



<2M -2 



7-2log(y'(l + 27)2 + (27tane)2) 

7+1 



7 



log V2T - 7-2 log(l + 27) j < (log \/21 - 2)2Af < 0. 



So on rf , if r] > T, < e' < 2 - log V21 and M large enough, 

|i^aAf(^,W^)|<^^^^(2M)l/3 

7 
X exp{-2(r/ - T)(l + -if'^{2Mf/^ 

+ ((log V2I - 2) - 2r(l + 7)"2/3)2M} 
-(7 + l)u; + l 1 



(7 + l)hw + 1 tt; + (07 - l)/((7 + l)(a + 1)) 

^g-2(ry~T)(l+7)'''^{2M)i/3g{logV2T-2+e')2A/ 

where e' is a positive number and e' <2 — log \/21- If -^'^ is large enough, 

27r 1 e-^/2 



g(logV2T-2+e')2Af ^ 



(53) 



2^37/(7 + 1)3 Ml/40 

g-2(r,-r)(l+7)V3(2Af)V3 ^ ^T/2g-r,/2^ 

and we get the result, since 
1 



(54) 



27r7 Jr^i 
4 



FaM{ri,w)dw 



27r wGrf □ 
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Lemma 3. If T is fixed and M is large enough, then for any r] > T, 



1 
27ri ^r^^^ 



FaMi'i],'w)dw 



< 



3M1740- 



Proof. For w £ remote ^^ denote I = ?li{w) = ^-^. Since arg(w) = ±|, 



we get by (52) 



3f? 



373 



2Mw^ + 2MRi{w) 



^2M|-(l±l>!, + l,„,('l + 2l±i, + 4('l±ir' 



r 



Then we take derivative 

df (7 + 1)', 1 



7 



7 



■log(l- 2(7 + 1)/ +4(7 + 1)^/^) 



dl 



l + -ioJi + 22±li + J2±l 

7 2 V 7 V 7 



7 



272^ 



■log(l- 2(7+1)/ + 4(7+1)^/ 



!ili_l)!p( 1 - (7 - l)^^l + 4( ^—^l 



,7 + 1, Y7 + I 



xi{l + 22±ll + 4(2±ll'' 

7 



X (1-2(7 + 1)^ + 4(7 + 1)2/2) 
and are able to find a positive number e" > 0, such that for < / < 2-^. 

-/2 

l-(7-l)(7 + l)/7^ + 4((7 + l)/70' 

(1 + 2(7 + l)/7/ + 4((7 + 1)/70^)(1 - 2(7 + 1)^ + 4(7 + 1)^^^) 
< 3e"/2, 

and on the two left-most points of T^cmotc (1 + \/3i)(2M)^i°/39 ^^^ (^^ 
V3i)(2M)-io/39, 



,(7+l)\2 



O/'J 



3f? 



(1 + 7)^ 
373 



2Mw^ + 2MR{w] 



«;=(l±V3i)M-io/39 



2M — 
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3 7^ 

(7 + 1)' 



(2M)-^°/^3 + 0{M 



-40/39- 



<2Af 



3 73 

r.(2M)-io/39 



(2M)3/i3(i + o(M-V39)) 



-3e"t^ dt. 



Therefore we know that for w € T^j^^^-g, 



3fi 



:i+7)^ 
373 



2Mw^ + 2MRi{wyj < 2M f -3e"t'^ dt = -2Me"l^, 



and have the estimation that ii rj >T, < e < e and M large enough 

[/>(2Af)-W39]^ 

|i^aAf(C,^)|<^^-^^(2Af)l/3 



7 



X exp<{ -(// - T) ^^ ' " — (2M) 

7 



<l±^(2A/)'/3, 

7 

e"/3^^(i+7)^(2Af)-2/3A2^^ 



7 






-(7 + l)«; + l 



1 



(7 + l)/7u; + 1 u; + (a7 - l)/((7 + l)(a + 1)) 

< g-(»7-r)(l+7)4/V7(2Af)i/"_e"'(2M)3/i3 

Now we get the result by inequalities similar to (53)^(54). D 
Lemma 4. // T is fixed and c is large enough, 



1 

27ri Jr 



-Tu+u'/^du 



< 



1 



Proof. Obvious. D 



Lemma 5. If T is fixed and M is large enough, then for any 1] > T, 
holds: 



1 /■ J. ( w (7 + l)(o + l) ,., ^ 

TT- / F^aAi [r], w) dw Ai(??) 

zvrz Jr^' 1 — 07 



< 



1 e-^/^ 
3 A^V40 ■ 
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Proof. On Vf^^^^, \w\ < 2{2MY^'^^ , so by (49) 

07 — 1 



X e 



7 

_(l+-^)4/3/^(-2M)l/3^^+(l+^)4/(3^3)2A/«;3 



(l + 0(M~^/39)). 



After the substitution u - 



(1+7)''/" 



{2MYI^w, we get 



FaAi{'n,w)dw 



(7 + l)(a + l) 
(«7 - 1) 



-VuW/3 ^^ 



(55) 



C(l+^)4/3/^(2A/)l/l 



X (l + 0(M-i/39)) 



and the 0{M ^Z^^) term is independent to w. 

On r°°, if r/ > T, e-(''-^)" < e^/^e-''/^. By (2) and (55), we have 



^ ., F.M(.,u.)d.-il±lli^i±liAi(,) 
27rz Jr,Al 1 - 07 



<eT/2g-r,/2 



(7 + l)(a + l) 



27ri(l — 07) Jr 



^"+"'/3|dn 



>{l+T.)4/3/^{2A/)l/l 



+ eT/2g-r,/2 



(7 + l)(a + l) 



27ri(l — 07) 

|^_T.W/3|^^0(M-V39) 

and we can get the result by direct calculation. D 



Conclusion of the proof of (43). Putting Lemmas 2-5 together, 
we get the convergence in L^ norm: 



(^ + l)2{M-JV)+l/3 



hm (1 + Q)2(^-M)-ia-2iV+ a7^^; '^_^^ '- ^^ _ „^)(2M)V3 



Af-+oo ■ ■ 7 

X e-2*^/(^+^)V27V-i(y)XT(r/) = Ai(7?)xT(r?). 
On the other hand, for r/ G [T,oo), 



(56) lim (1+7 



M^co 



1^2{N-M)-l^M-NyM-N+l/2^{l-y)/{l+'y)My 



and 



(57) (1 + ^-l)2(A^-J\/)-leJ\/-A^yA/-iV+l/2g-(l-7)/(l+7)A/?/ < 1 + O f ^^) . 
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Therefore, in L? norm, 

= Ai(??)x(r/). □ 

Now we conclude the proof of the — 1 < a < 7"^ part of Theorem 1. By 
Stirhng's formula, we get 

(58) hm (2M)^(*^-^) 1^^ ~ 1)! 2(7V-M) 4JV-1 ^ ^ 

^ ' M~*oc> ' (2M-1)! ' 

and then by (34), (38) and Proposition 5, we have the convergence in trace 
norm 

i^c (1 + 7)2/3 x{i)S.,{i.v)x{ri) 
(59) 

= ix(e) (Ai(e) Ai(r/) + Ai'(e) j^ M{t) d?j xiv), 

which implies that in trace norm, 

lim xiOSibi^, r])x{v) = 0- 
Now we get the desired result 

lim x(6'S*4(C,??)x(^) = lim x{0S4a(.S,,v)x{v) =x{0S4{^,v)x{v), 

Al—*oo M^oo 

and in the same way 

lim x{0SD4{^,v)x{ri)=x{0SD4iC,r^)x{v), 

Ai— ♦CO 

lim x{0lS4{^,v)x{v) =x{0lS4{^,v)x{v)- 
Therefore, in trace norm 

lim PT(e,r/)x(r?)=x(e)f§^f;^\ ^^f'!(^]x{v), 

and the convergence of Fredholm determinant follows. 

3.2. Proof of the a = 'y~^ part of Theorem 1 . When a = 7"^, the 1 — 07"^ 
in (59) vanishes, so we need other asymptotic formulas for tp2N-i{'n) ^^^ 
V'27V-i('?)- '^1^^ approach is similar to that in the a < 7"^ case, so we just 
sketch the proof. 
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Proposition 6. For fixed 7>1, = 7 ^ , e >0 and any T, we have the 
convergences in Lp' norm with respect to ^ or tj: 



M— >oo 



X {2Mx)x^'-''-'r^e-'''''x{i) 



M-N-l/2^-Mx^ 

e'^AiiOxiO, 



M^oo 



(60) 



-eri 



A[{t)dtx{v), 



lim (1 + a)2(^-^^)-ia-2^+ie^^-^7"'^+'e-^^27V-i(y)x(r?) 
= e-'^s^'\v)xiv), 

= e^«Ai(C)x(0- 

Sketch of proof of (60). We perform the same algebraic procedure 
and use the contour f = Ti U f 2 U f 3 U t^ which is slightly different 
from the F''^ in the a < 7^^ case (see Figure 4): 

e/2 



Ff = (4 - t 



7 + 1 



0<t<4 



(1 + 7)1/3 



173(2^^^)" 



-1/3 



1 2 



nM 



^M 



76/2 



(1 + 7)4/3 



(2M) 



-1/3 tTTJ 



^^_g5^i/3 

7 + 1 



e/2 



(1 + 7)1/3 



1 5 

3 - - 3/' 



(2M)-^/'^<t<4^, 



7 



7 + 1 



+ it 



-2V3—^ < t < 2VS—^ 
7+1 7+1 



and for asymptotic analysis, we define F^j^^^-g, F^^^^^j, F^^ and F^^ in the 
same way as (50)-(51). Then we get 

^2;v-i(y) = -(l + a)^('^-^)+^a2^-i 

„2M 



7 



,(2M)/{l+-y)y 



(7 + l)2(M-iV)+l 

1 ^ ^ ii±ll!^r.A^^V3, 



X (p_ exp 

27ri Jr^^ 



7 



-(2M)^/'^7?u; 
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„ 2Mw'' + 2MRi(w) 

— (7+ l)w + 1 dw 
(7 + l)/^w + 1 w 

If we denote 

r (1 + ^)4/3 (1 + 7)^ 

FM(r],w)=exp\ -^—^ — (2MW^7]w+ ^ 1' 2Mw^ + 2MRi(w) 

3^d 



7 

^^ -(7 + l)^; + l 1 
(7 + 1)/7W + 1 w' 

then parallel to Lemmas 2-5, we have: 



Lemma 6. For any T fixed, and M large enough, if r] >T, then 



-er] 



1 
2Tri JfM 



FM{v,w)dw 



< 



1 e-^''/2 
3 MV40 • 



Lemma 7. For any T fixed, and M large enough, if t] >T, then 



-er) 



T— /_ FM{r],w)dw 

2m JfM ^ 



< - 



1 e~^''/2 



3 MV40 • 



Lemma 8. IfT is fixed and c is large enough. 



1 f Tu+u^/3^ 



27ri Jrs° 

>c 



1 
< -. 

c 
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Lemma 9. For any T fixed, and M large enough, if r] >T, then 



-£J7, 



1 

27ri JfM 



FM{ri,w)dw-e-i''^s^^\ri) 



< 



1 e-^''/^ 
3M1740- 



Using Lemmas 6-9, we get the convergence in L^ norm: 



lim (l + a)2(^-^^)-^ 






7 



2A/ 



■■y)y 



A/^oo 

xe~''iip2N~i{y)x{^) 

Furthermore, because of the Umit result (56) and (57), we get the L^ con- 
vergence 



hm (1 + a)2(^-^^)-ia-2^+S-''^+'e*^-^e-^>27V-i(y)x(r?) 
M— »oo 



D 



Now we conclude the proof of the a > 7 ^ part of Theorem 1. Using (34), 
(58) and Proposition 6 we have the convergence in trace norm 

lim xiOe'^S^b{tv)e-'''xiv) 



M^oo 



ix(e)e^« (Ai(Os(^) iv) + Ai(0 1" Ai(t) d?j e-'^xiv) 
xiOe'^AiiOe-'Mv), 



and this together with the conjugated convergence result (discussed in the 
Appendix) of S^aiCv) ^^ formulas (39) and (40) of Section 3.1 conclude 

(61) lim xiCy^M,v)e~'''xiv) = xiC)e'^^i{C,v)e-'''xiv)- 

In the same way we get 

lim x{CV^SD^{C,rj)e'^xiv) = xiOe'^W4^,7^)e'\{r^), 

hm x(0e"'^^4(e,r?)e-^''x(r?) = x(e)e-^«^4(e,r/)e-^''x(r/). 

M— >oo 

Then we get the convergence in trace norm of a conjugate of -Pt(^,??) 
,Je'^Si{C,r])e-''^ e'^SDi{C,r])e'A ,. 

= vfn ( e^«l4(e,r/)e-^ e^m,{i,r^)e^A 
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and the convergence of Fredholm determinant follows. 

3.3. Proof of the a > "y^^ part of Theorem 1. If a > ^~^, the location as 
well as the fluctuation scale of the largest sample eigenvalue is changed. We 

change variables as pM = {a+ 1)(1 + :^) and qm = (a + 1)0"^^^^-^, 
and then by (36) the kernel S^:{x,y) after substitution is [here * stands for 
4, 4a or 4b, and the S^:{S,,r]) in this subsection is not identical to that in 
Sections 3.1 and 3.2] 



S4^,v) = {a + l)Jl ^ ^ 



7^a^\/2M 



(62) 

X '^*(^'y)lx=(a+l){l+l/(72a))+{a+l)^l-l/(72a2)l/{V2M)g ' 
y={a+l){l+l/{j2a))+{a+l)^/l-l/{'y^a2)l/(V2M)v 

We analyze S^biCv) first. 

Proposition 7. For fixed 7 > 1, a > 7^^, e > and any T, we have 
convergences in L^ norm, with respect to ^ or tj: 

{rJl„ I 1 ^A/-A^+l/2 I 



(63) 



M-^oo (72a)M+7V+l/2(Q _^ i)M-JV-l/2 
X g(7'a2-l)/((72a+l){a+l))Mx 

X e^«L('^^-^» (2Mx)x^^-^"i/2e-*-^-x(C) 



(64) 



V2^' ^l 4 (7^0 + 1)2 ^ +e4|Xi4j, 

hm 1 (7^a + l)^^^-"-^/^(7^a^-l) ^n^j^.^^^ysl.^u^. 

^ g(72a2-l)/({72a+l)(a+l))A/j/ge»7 

X /" i^f-i""^" (2Mi)t*''-^-^/'e-^^* dt x(r/) 



J/ 



_2 -l/4(74a2+72a2 +4720+72+1)7(72 a+l)2r)2+£r? 



X{r\), 



J2^ N A/-7V+1/2 



Af^ooV(72a + l)(a + 1) 
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^g-l/4(72a2-l)(72-l)/(72a+l)2r,2_e^^^^^^ 



A/^ooV(72a+l)(a + l)y {-f'^a?-l)M 

We only prove (65). The identity (45) still holds, but we need to use 
another contour and a new procedure of steepest-descent analysis. 
Since 



g-2Mj/2 



{z + lf^ 



^2N 



_ „2M(-(a+l)(l+l/(72a))2+log(2;+l)-7-2 logz)-{a+l)^l-l/{-f^a'2)V2M'nz 

if we denote (ignoring the ambiguity of values of logarithm) 

g{z) =-{a+l)(l + 4^") z + log(z + 1) - 7"^ log z, 

then we get: 

• 9'{z) = -(a + 1)(1 + :^) + ^ - V, with zero points z = -jip^ and 
z = ° ■ 

(l + a)2(^-l)<0. 

So we take z = — . ^ as the saddle point, and locally around that point, 
after the substitution w = z + jtzt^ 1 we get 

+ w] = — 2 ^ log(7^a) - (1 - 7~^) log(7^a + 1) + 7~^7ri 



1 + 7^a / 7^a 

+ ^(7-'+7«)'(i-:^)^«' + ^2(^) 

where 

R2{w)=0{w^) asw^O, 
so that 

^2M{~yz+\og{z+l)^-f-'^ log 2) £ ^^ 

C {{a + l)z + a){z + l) 
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(67) = / exp(2Mf^^ + log(72a)-(l-7-2)log(7^a + l) 



+ 7 ^7ri + -(7 ^ +-iafil-^^-^]w'^ + R2{w) 



(a + l)Wl 



1 T] 



1 



W 



I'^a? y/2M\ 7^0 + 1 
w- 1/(720 + 1) 



((a + l)w + {-f^a^ - l)/(72a + l)){w + {l^a)/{-i^a + 1))) 

„2„\2Af-l 



dw 



{I'a) 



- + 1 (72a + 1)2(J^^-^) 



„2A//(72a+l)x 



+ ^ (7~^ + 7«)^ (1 - ^) 2Mt/;2 ^ 2Mi22 (w^) | 



(0^aTI)7(7^«)«^TI 



■dw, 



w; + (72a2 _ l)/((72a + l)(a + 1)) 

where S*^ is a contour around t^tt, composed of S*^, Sg^, Sg^ and £4^, 

which are defined as (see Figure 5) 

Sf = {-it\-2 < t < 2}, Sf = {4 - t + 2i|0 < t < 4}, 

Sf = {4 + ft|-2 < t < 2}, Sf = {t - 2i|0 < t < 4}. 

And for the asymptotic analysis, we define (see Figure 6) 

yM _ < p vA/|| I ^ /Vf-2/5T. yAf _ yM \ yM 

^local — 1^ fc ^ WWlS-lV-i I, l^i-cmotc — ^1 \ ^locab 




Fig. 5. E'' 
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y^ 



Fig. 6. E° 



S°° = {-it|-oo<t<oo}, J:^^ = {w£T,°^\\w\ <c}, 



S5=^ = s°° \ s°° 



<c- 



Then if we denote 



GaM{v,w) = i'y ^+7a)Wh j^ 



2M 



X exp<i -(a+ 1) J M - ^-^ j2Mr?u; 

+ ^(7"^ + laf (l - -^]2Mw'^ + 2MR2{w] 
2 \ 7^a^/ 



-(72o + l)u; + l 



1 



(72a + l)/{^^a)w + lw + (72a2 - l)/((72a + l)(a + 1)) ' 
we have four lemmas similar to Lemmas 2-5: 



Lemma 10. For any T fixed, and M large enough, if r]>T, then 



-^v. 



1 



27ri Jsf usf usf 



GaMiil,w)dw 



< 



1 e-^'' 
3MV10- 



Lemma 11. For any T fixed, and M large enough, if 7]>T, then 



e ''''-— / GaM{r],w)dw 

2m Jt,^i ^ 



1 e""'' 
< -- 



3 Ml/10 • 
Lemma 12. IfT is fixed and c is large enough, 

J_ j g-(a+l)/{7-^+7a)T«+«V2^^j 

27ri Jr°° 



1 
< -. 
c 
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Lemma 13. For any T fixed, and M large enough, if r]>T, then 
e"^''-— / GaM{ri,w)dw 

local 

(j^a + l)(a + 1) ^_i/2((^(<,+i))/(^2„+i)^)2 _ ^^ 



(72a2 - l)y/2TT 

1 e-''' 



< 



3 Ml/10- 



Their proofs are the same as those of Lemmas 2-5, and we need the 
identity 



— / e"("+^)/('^~'+'^")''"+"^/^(in = —e~^ 

2m is°° \/27r 



/2(7{a+l)/{72a+l)r,)2 



Sketch of proof of (65). Because the pole z = —-^, which is w = 
— ( 'P i°i\7 1 1N in the w plane, is not in side of S^^, so 

(68) / e-2^^-^^ii^^^^(iz = 0. 

Jc z + a/{a + l) 

Similar to but subtler than (56) and (57), if we denote (here we have a 
notation conflict with the rj defined in Section 2.3, but there should be no 
confusion) 

/ -,2„ \ M-N+1/2 



X g-(7^-l)a/((72a+l)(a+l))Afy-er, 



we have for r] ^ [2^,00) 



lim rA/(r/) = e 



-1/4(72 a2-l)(72-l)/(72a+l)2r,2-£r, 



and for a large enough positive C, rj £ [C, 00) and Mi < M2, pointwisely 

rMAv)>rM2{r])>0, 
so that we can use the dominant convergence theorem to prove that in L^ 



norm. 



lim rM{r,)x{v) = e-i/<^''^'-i)(^'-i)/«^'"+i)')'''-^'''x(^). 



Finally, since from (45), (67) and (68), 
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(72a + l)2(A./-7V)+i ^(^^a^ _ i)2M 

1 



X yM-^+l/2e{l-7^a)/(l+7^a)Af,_ / GaM{r,,w)dw 
yM~N+l/2^~-({'y^-l)a)/{{'y^a+l){a+l))My 



,(72a2-l)/(72a+l)(a+l)Afy 



+ (l + a) 



2(M-Ar) 2Ar-l_ 



(7^a 



2^\2M 



X e 



(72a + l)2(M-7V)+i ^(^a^ _ i)2M 



-(72a2-l)/((72a+l)(a+l))AfyJ_ 



27rz Jy,m 



we get 



7^0 



(72a + l)(a + l) 



M-N+l/2 



X e 



Af-TV (72a2-l)/((72a+l)(a+l))A/y^-£,? 



V'27V-i(y)x(r/) 



rM{v) 



l + (l + a) 



2(A/-Ar) 2Af-l 



a 



(7^a 



,2„\2Af 



(^2^ + l)2(M-iV)+l 



1 



V(72a2-l)2M''''Pr(72a + l)(a + l)^^^^/ 



X -— / GaM{'n,w)dw 



xiv), 



and get the L^ convergence 



7^0 



lim , , „ 

Af^ooV(72a + l)(a + l) 



A//-Ar+l/2 



X e 



Af-TV (^2^2_l)/((^2a+l)(a+l))A/y^-e,; 



'4'2N-i{y)xiv) 



= lim rA/(?7)x(^)=e-^/^(^''^'^i)(^'^i)/(^''^+^)''''-"''x(r/), 
Af^oo 

because for a > 7"^ and r] G [T, 00), we can verify by by elementary but 
tricky estimation that 



lim (1 + a 

A/^00 



,2{Af--Af)^2Af-l 



il'a 



2„\2M 



-(72a2-l)/((72a+l)(a+l))2Afy 



(72a + l)2(M-7V)+i ^(7^a2 - l)2M 
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uniformly, and by Lemmas 10-13, in L^ norm 

lim e"^''-— / GaMir],w)dwxir]) 



(j^a + l)(a + l)_^_i/2(^(a+l)/(72a+l),7)"-£'?^(^). 



(72a2 _ 1)^2^ 



n 



For notational simplicity, we denote functions on the left-hand sides of 
(63)-(66) by Fi(Ox(0, ^2(r?)x(r/), F-i{7])x{v) and F^iOxiO, and denote 



CM = (2M)2(^^-^)|^-l|e2(^-^^)7^^-^ 



By (58), we have 



lim Cm = 1- 

M^oo 



Then we get from (34), (62) and (63)-(66) 

5,,(^,^) = _£^(e-(7^-^-i)/((7^-+i)(-+i))An--?^)--«-^)Fi(e)F3(r?) 
(69) ^ 

+ e(7'«'-i)/({7'"+i){«+i))A^(^-?/)+f(C-^'?)^^(^)^2(r?)). 



If we define 



N-2 , 



SD4a{x,y)= J2 —(V'2j(a^)^2i+i(2/)-V'2j+i(^)V'2j (?/))> 



Af-2 , 



ISia{x,y)= Y^ —{-llj2j{x)lp2j+l{y)+i^2j+lix)Tp2j{y)), 

and 



i=0 ^J' 



like 



SD4h{x,y) = {lp2N-2ix)lp2N-l{y) - '^2N~l{x)'4^2N~2(.y)), 

rN-1 

ISib{x,y) = {-^2N-2{x)lp2N~l{y) + '4'2N-lix)'lp2N~2iy)), 



N-2 -, 
54a(x,y)= Y^ —{-ij2j{x)i^2j+l{y)+i^2j+l{x)lp2j{y)), 

S4b{x,y) = {-'ll)!2N-2(.x)i^2N-l{y) + 1p2N-l{x)'4'2N~2(.y)), 

rN-1 
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in (28) and (29), and by (35) and (37) define [* stands for 4, 4a or 46] 

X '5'^*(2;!?/)lj.=(„+i)(i+i/(^2„))+(„+i)^i_i/(^2a2)i/(^/2M)^' 
2/=(a+l)(l+l/(72a))+(a+l)Vl-l/(72a2)l/(v/2M),, 

y={a+l){l+l/{'y^a))+{a+l)^l-l/{y2a'^)l/{V2M)r, 

like 



5*(e,r/) = (a + 1)^1-1/(7^^2) 



/2M 



X 'S'*(a^>y)l3.=(a+i)(i+i/(^2„))+(a+i)^i_i/(^2a2)i/(^/2M)5 
y=(a+l)(l+l/(72a))+(a+l)v/l-l/(72a2)l/(v^2M),y 



in (62), then in the same way of (69), we have 



^^,(^,^) = £^C'M(e-(^''^'-^)/«^''^+i)('^+i))^^(^-^)-^«-'')Fi (0^4(77) 
IS^,{^,7^) = ^(e-(^'«'-i)/((^'-+i)(«+i))*^(-J/)--(5-'?)F2(OF3(??) 

_e(7'«''-l)/{(7'«+l){«+l))A^(^-3/)+£(C-'?)i73(^)i?2(7y)), 

with 

(^2^2 _ l)3/2^/2M 



07(7^0+ 1) 

Now we write Pt (■?>??) as the sum 

PT{^,rj)=PTa{C,r])+PTb{tr]), 
with 
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If we denote 



CMF^ii) 



2 F3(0 
e 







g(72a2-l)/((72a+l)(a+l))Af(a;-xo)+e? 



(72a2-l)/((72a+l)(a+l))A/(x-xo)+e5 



-I , p -{7^a2-l)/((72a+l)(a+l))M(j/-j/o)-£r, 



CM-F4(6g-(72a2-l)/((72a+l)(a+l))M(j/-j/o)-£« 



2 ^3(6 

e 
with 



(72a2-l)/((72a+l)(a+l))M(y-yo)+£« 



a^o = yo = (a + 1) ( 1 + ^) + (a + 1)W 1 



then we have the result of kernel conjugation 

with the entry 

U{i)pTb{i,r,)U-\ii)2i 

Cm 

We want U{^)PTb{C,'n)U~^{'n) to converge in trace norm as Af ^ oo, and 
need the results: 

Lemma 14. /n trace norm, 

hm C/(OPT6(?,^)f/-'(r?)2i = 0. 
M— >oo 
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Lemma 15. In L? norm, 

=-7i^^"n-4 p^n? ^ +^^F^^)- 

The proof of Lemma 14 is obvious. The main ingredient in the proof of 
Lemma 15 is (64) and the fact that i^4(C)/-f3(0 approaches to 1 uniformly 
on [T, oo). 

We need another convergence result on U{^)PTa{i^'nW~^{Ti])'- 

Proposition 8. In trace norm, 
(70) hm C/(C)Pra(e,r/)C/-'(r/) = 0. 

The proof is left to the reader. Since all the four entries in PTa{5,,rf) can 
be expressed by Laguerre polynomials like (32) and (33), the asymptotic 
results like (63) and (64) give the convergence (70). 

By Lemmas 14 and 15 and Proposition 8, we get in trace norm 

hm det(/-PT(C,r?)) 

M— >oo 

= hm dei{I-U{i)PT{i,il)U~\ii)) 

M— >oo 

= hm dei{I-U{i)PTb{i,v)U~\il)) 

oo V 2vr 
and we get the proof of the a > 7"^ part of Theorem 1. 

4. Proof of -Fgsei = -fcoE- In manipulation of kernels, we follow the 
method of [30]. The procedure seems informal and cursory, but is carefully 
justified in [30]. 

For notational simplicity, we denote [xiO = X{t,oo){0] 



poo 

5(^) = l-sW(0= / Ai{t)dt. 
First, we express the integral operator 

Y(^)T(e v)Y(n) = ( ^(^)£4(?,^)x(^) xi0^4iC,v)xiil) 

\x{0iS4{C,v)x{v) x{0S4{v,C,)x{v) 
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by 



x(0/ V-^'5'4(?,^)x(r?) 'S'4(ry,^,)x(^). 



since by (22)-(23) and taking limit, 

— IS4{^,r]) =S'4(^,r/), 

Then using (21) for A bounded and B trace class, upon suitably defining 
the Hilbert spaces our operators A and B are acting on, we find 



d 







ISi{i,7])x{ri) S4ir],0xiv) 



det I /- I ^^^^dC , ,— 

x(0/ V-^'5'4(?,»?)x(^) 'S'4(r?,^,)x(r/) 

det I /- I £4(e,^)x(^) J4(^,e)x(^) \ [ Xir])^ 
,/54(e,r/)x(r?) 54(r/,e,)xW ' ' ^ 







x(??). 



:det 



I 



= d - 

ISA{i,r])x{ri)— S4{7],C,)x{r]) 

1 0\ 



V 



V- 



7 



and by conjugation with [-^ ^), we get 



d 



det I 



^'54(^, v)xiv)-Q- + 'S'4(??, Ox(??) 'S'4(??, Oxiv) 











det(^/- (^/54(e,r?)x(r/)|^ + 54(r?,0x(r?) 



Since 

oo 



IS4{^,r,)i-f{v)dr^ = IS,{^,v)f{vrr,ZT 



d_ 
drj 



IS^{tv)fiv)dv, 



d_ 

IT drj 

as an operator 

^4(e,r/)x(r?)|^ =^4(e,oo)5oo(^) -^4(e,T)<5T(7?) - -^1^4{tv)xiv), 

where 6oo and 6t are (generalized) Dirac functions. Then with the help of 
identity 



/•oo /"OO /"OO /"OO 

y KAiry{t,v)dt + J KAiry{^,t)dt = J A[{t) dt J Ai(t) 



di, 
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which can be proved directly from (1), we get 



I-{ KAiry (^, V) - i,BiO Ai(77) + Ai(r?) ) xiv) 



+ 



^ J^ AAiry (e, t) dt - \b{T)B{0 -\b{0 + \b{T)^ 5t{v) 



+ \B{i)5oo{ri). 

Now we denote R{^,r]) as the resolvent of KAiry(C,??)x('?)) such that as 
integral operators 

(71) / + R{C, r]) = {I- i^Airy (e, V)x{v)r\ 
then 

I- (lB,{^,7i)x{v)^+^4{v,Oxiv)) 

= iI-KMry{C,V)xiv)) 

x(/-(/ + ii)(l-^i?(e))Ai(r/)x(r?) 

- ii?(r)i?(0 - ^5(0 + ^^(r))<5T(r/) 

Again by the formula (21), in the form of (formula (17) in [30]) 

(72) det I 



; I 7 - ^ Ofc /3fc j = det{6j^k - ioij,Pk))j,k=i,...,n 



we get 



det (/-(/ + R) (l - ^B{0^ Ai(7?)x(r?) 

+ {I + R) Q J^ i^Airy (e, t) dt - \BiT)B{0 - Ib{0 + \b{T)^ St{v) 



+ ^(I + R)BiOSooir]) 
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1 + aii ai2 ai3 

: det ( a2i 1 + a22 "23 

"31 "32 1 + "33 , 
where upon the definition 



oo 



we define 
an = ((/ + /?)(! -ii?(0),-Ai(0)r, 

-Ai(e)^ 

a2i = (/ + i?)(l-ii?(0)l5=T> 

«22 = (/ + i?) (^ J^ /^Airy (C, i) o!t " \b{T)B{0 " ^i?(0 + ^i?(T) 

a3i = (/ + i?)(l-iB(0)lc=oo = l, 

"32 = (/ + i?) (^ 1^ KMryit t) dt - \BiT)B{0 " ^i?(0 + ^^^ 



/ T 



C=r 



5=oo 



a33 = 5(/ + fl)-B({)k=oo = 0. 

If we take elementary row operations, we get 
/1 + aii ai2 ai3 

det 021 1 + "22 "23 

\ asi "32 1 + a33 , 

/l + aii-ai3 ai2 - |S(T)ai3 "13 ' 

= det 021-023 1 + 022 - ^^(?')a23 "23 

V 1 

where 

/3ii = ((/ + i?)(l-B(0),-Ai(0)T, 



i=T 
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/3i2 = ^^(/ + R) (^j^ KAiryiC, t) dt - B{T)B{0 - B{0 + B{T)^ , - Ai(e) 

P2i = iI + R){l-B{0)\^^T, 

P22 = l{I + R)(^J^KAny{C,t)dt-BiT)B{0-BiC) + B{T)^ 

Using (71) and (72), we observe [s^^HS.) = 1 - B{^)] 
det{I-KAiry{^,v)x{v))det 

= det ^I - (KAiry (C, V)X{V) + s^'^ (6 Ai(7?))x(r?) 

+ ^(^I^KAiry{C,t)dt-B{T)B{0-B{0 + B{T)yT{v) 

If we denote R{S,,ri) as the resolvent of {KAiry{£,,v)x{v) + ^ iO -^K'n))x{v) ^ 
so that as operators 

/ + m r,) = {I+ (i^Airy(e, V)X{V) + s^'^ (6 Ai{r,))x{v)r' 
and 

Q{0 = {I + R)I^J^KAiry{^,t)dt-B{T)BiO-B{0 + B{T)y 

then 

Fgsei = det(/ - (KAiry(C, r?)x(r/) + s^'\0 Ai(r?))x(r?)) det(7 + lQ(e)<5T(??)). 
To prove Theorem 2, we need only (6) and 

det{I+lQ{C)6T{v)) = l, 
which by (72) is equivalent to 

(73) Q{T) = 0. 
If we take /(O = Q(0 + 1, then (73) is 

(/- (KAi.y(e,r?)x(^) +s(^^(e)Ai(r?))x(7?))(/(e) - 1) 

= / KAiry{^,t)dt-B{T)B{0-B{0 + B{T), 
Jt 

which is equivalent to 

(74) iI-iKAiry{tv)xiv) + s^'\OMv))x{v))fiO = s^'\0- 
The integral equation (74) is solvable, and the solution is 

(/+i?)s«(e) 



fio 



i-((/+i?)s(i)(o,Ai(e))T' 
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Therefore to prove Theorem 2 we need only to prove /(T) = 1, which is 
equivalent to 

(/ + i?)s(^) (T) = 1 - ((/ + fi)s(i) (0, Ai(0)T- 

This is a nontrivial result, but it can be derived by results in [30]. In Section 
VII of [30] Tracy and Widom define function q and u for both GOE and 
GSE. Our (/ + R)s^^'{T) is equal to \/2 times their q in GOE and our 
((/ + i?)s(^)(6iAi(0)r is equal to 2 times then u in GOE. With 

((/+ i?)s«(e), Ai(e))r = 1 - e-r^W'^^ 

where q is the Painleve II function determined by the differential equation 

q"{s) = sq{s) + 2q^{s) 

together with the condition q{s) ~ Ai(s) as s ^ oo. 

We can give a proof of (75) and (76), based on the method and results in 
[29]. First, assume T is fixed, then (/ + R)s^^' is a function, and we have 



(75) 
(76) 



d 



{I + R)sW{^) = {I + R) 



&(^)(g) 



+ 



d_ 



,{i + R) 



.w 



(0- 



Since 4s^^\0 = Ai(^) and we have (2.13) in [29], which is 



d_ 






(2 + R) Ai(e) • (1 - Ky{Ai{ij)x{v)) + R{V, T) ■ p{T, r,), 



where p{x, y) = 6{x — y) + R{x, y) is the distribution kernel of 1 + i?, and K^ 
is the transpose (as an operator) of KAiry(C)'^)x(^)) we have 

^(/ + i?)s(i) io = {i+R) Am - (1 + ^) Ai(e) • ((/ + R)s^'^ (6, Ai(e))r 

+ R{tT)-{l + R)s^^\T). 
If we regard T as a parameter, then we have 



(77) 



^(/ + i?).«(e;T) 



-R{C,T)-{1 + R)s^^\T) 



because (2.16) in [29] gives 



— {l + R) = R{^,T).p{T,v). 

Therefore, if we set £, = T and take the derivative with respect to the pa- 
rameter T, we have 



d 
dT 



((i + i?).«(r)) 



| + A)((, + ^).a)(^)) 



(1 + R) Ai(r) • (1 - ((/ + i?)5^^^(0, Ai(O)r)- 



5=T 
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On the other hand, by (77) we have 

= _(i + i?),(i)(r) . Ai(r) + (^(/ + i?).«(0, Ai(o)^ 

= _(i + i?)s(i)(T).(^Ai(r)+^"i?(e,r)Ai(e)de^ 

= -{l + R)sW{T)-{l + R)Ai{T). 
(1.11) and (1.12) m [29] give the result 

{l + R)Ai{T)=q{T), 

and now if we denote (I + i?)s(i) (T) = st and ((/ + ii)s(i) (0, Ai(0)T = wt, 
we have 

— ST = q{l-WT) 
^—{l-WT)=qST- 

Now we can get (75) and (76) by boundary conditions. 

APPENDIX: DISCUSSION ON THE TRACE NORM CONVERGENCE 
OF INTEGRAL OPERATORS RELATED TO LUE 

For convenience, we write (44) as 
(78) Lf (^■^-^»(2My) = '-^i e-^'^^y^ ] ' dz, 



^ ^ ^' 2m Jc (z-l)J+i 

where C is a contour around 1, and we have another integral representation 
of Laguerre polynomials 

Am-N)), . ^ {2{M-N)+j)\ 1 

(79) 

where D is a contour around 0. 

Recall the integral operator K(x,y) [11] for the rescaled LUE with pa- 
rameters 2N and 2M, and by (78) and (79) we have 

2N-1 ., 
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(80) ^ ^ 



(2^i)2 x^ . ^.^^ 






^2{M-N)+3+l ■ 

We can write the sum of integrands in (80) as 

27V-1 

e~ 



E2Mxz ^ -2Myw \^ ^) 

{z- 1)J+1 ^2{M-N)+j+l 



^2Mxz -2Myw 



^2(M-N) 1 2^1/^(y;_l 



e"""'~e —'"- — ^^^^^ — ^^- ^ — > 



w'^i^-^) {z-l)w ^^\{z-l)w 



J 



2N 



2Mx. -2Myu, ^^^"'"^^ 1 1 - (^(^ - l)/((^ - 1)^)) 

W2{M~N)^Z-1)W 1-Z{w-1)/{{Z-I)w) 



_2 2Mxz 2{M-N) -2Myw _ ^ 



Z-W y;2{Af-7V) 

_ -*- 2Mxz ^ „-Myw \^ ^) 

Z-W (z-l)2^ u;2A/ • 

By the residue theorem, let C and D be disjoint, then for the variable z, the 
pole z = w is outside of C, 

On the other side, we assume K(w — z) to be less than 0, and get 

2M / e*2A'^('"-^) dt, 



Z — W JO 

so that we have 



(27ri)2 Jc Id z — w {z — 1)2^ u; 

(81) = Ml! I dzi dw r e-2^'^fe+*)- ^'"^ 



2A/ 



(27r«)2 Jc Jd 7o (-2-1 



2N 



/•oo /I /• ^2M 



2Ar 
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2N 
xi^^i e"^'-A^+*^"' ^"^ ,:;; dw ] dt. 



fj_ I J2M{x+t)w {W-I 
\2TTilD U;2M 

Put (80)-(81) together, we get the resuh 

M-N My 

K{x,y) = -{2Mf y^^^^ 



fj_ I ^2M(x+t)w i^ - 



V 2vri Id w^^'^ 



(82) X (^f 6''"'^''+')'" ' ', dw 



x(—l e-^M(y+t)z ''^'' dz\ dt. 

\2TriJc (z-l)2^ J 

To find the probabihty that the largest eigenvalue > T in the LUE, we 
need to consider the integral operator from L'^{[0,oo)) to ^^([0, oo)) with 
the kernel x(,^)K{x,y)xiy)- We can decompose it into the product of two 
integral operators by (82): 

x{x)K{x,y)xiy) = -i2M)\{x)J{x,t)x[o,oo)it) ox[o,oo)it)H{t,y)xiy), 

where x{x)J{x,t)x[o,oo){t) and X[o,oo){t)H{t,y)x{y) stands for two integral 
operators with these kernels, and 

j(x t) - i — i c^M{x+t)w jw - if^ , 

Hit, y) = .-"-e-^^ £ e--(^-*)^^^i^ d.. 

Since we consider the limiting distribution of the largest eigenvalue around 
(1 + 7-1)2, we take p= (1 + 7-1)2, g=li±^^7^, x = p + qi, y = p + qr] and 

t = qr. Then for the rescaled kernel x{i)K{^,rj)x{r]), we have 

x{€)K{i, v)x{v) = xiOJi^^ 'r)X[o,oo) [t] o X[o,oo) {t)H{t, v)x{v), 
where 

■^i^'^i- ^ ^^^ x^i-^e^^-2TTiJD w;2M «'^' 

^^'^^ 7 72^e^-M 27ri Jc (^ _ 1)2^ "^^- 

We want to prove the trace norm convergence 
lim x{i)K{^,r])x{r]) 

M^oo 
(83) =X{mAiry{C,V)x{^) 

= xiO Ai(C + T)x[o,oo){r) o X[o,oo)(t) Ai(T + r])x{r])- 
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By results in functional analysis, we need only to prove the convergence in 
Hilbert-Schmidt norm of (e.g., see [20]) 

(84) ^im^x{OJitr)xio,oo)ir) = xiOKf Airy it v)xin), 

(85) lim X[o,oo)ir)H{T, ri)x{r]) = X[o,oo)(t) Ai(r + r?)x(r/). 

Since for integral operators, the convergence in Hilbert-Schmidt norm is 
equivalent to the convergence in L^ norm of their kernels as two variable 
functions, we can verify (84) and (85) by asymptotic analysis similar to that 
in Section 3. 

For the integral operator xiCjSAaiH^ffjxiv) ™ (39), we have 

1 1 ~ ~ 

x{i)SAai{i,rj)x{r]) = ^x{O^J{tT)X[o,oo){T-) ° X[o,oo){T)y/yH{T,'n)x{r]), 

where we define J and H in the same way as J and J, but use parameters 
2N -2 and 2M - 2 instead of 2N and 2M. Similarly, in the Siai part of 
(61), we have 

x{Oe'^S,ai{tv)e^'''x(.r]) 

1 e^^ ~ /y ~ 

= 2^iO^J{t'r)X[o,oo){T-) o X[o,oo){T-)-^H{T,'r])x{v)- 

We can give rigorous proofs to (39) and (61) in the same way as (83). 
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